feat(project): add LLM-as-a-judge and code-based evaluator TUI wizards by tejaskash · Pull Request #2468 · aws/agentcore-cli

tejaskash · 2026-09-30T00:32:39Z

Description

What. A bare agentcore add evaluator llm-as-a-judge or agentcore add evaluator code-based on a TTY opens a wizard. add evaluator joins the add menu as a submenu.

llm-as-a-judge: name, level, model (Bedrock or OpenResponses, each with its own model ID input), instructions, rating scale preset, review.
code-based: name, level, Lambda (scaffold a new one with a timeout step, or use an existing one by ARN), review.
Each handler and its wizard share one input builder, so both refuse the same input with the same message.
RouterScreen now keeps command names in one column when a description wraps.

Why. These were the evaluator entries in the add wizard design. The wizards ask only what has no default. Description, KMS key, tags and inline rating scales stay flag-only. Instruction placeholders are not checked locally, following #2124, which leaves that check to the service.

The Bedrock judge defaults to global.anthropic.claude-sonnet-4-6. The Evaluator service sends temperature: 0.0, and Claude Opus 4.7 and every later Claude model refuse it. I checked every active Anthropic profile in us-west-2.

Related Issue

Closes #

Documentation PR

Not applicable. command.md is regenerated for the two shorter descriptions.

Type of Change

New feature

Testing

bun test (3729 pass, 0 fail), typecheck, lint:check, format:check, build
Drove both wizards through the TUI harness in a real project, covering every step, each validation message and back navigation
Deployed both wizard-built evaluators to us-west-2. Both came back ACTIVE, then the stack was removed through the CLI

`agentcore add` now lists `evaluator` with a submenu for its two leaves, and a bare `add evaluator llm-as-a-judge` or `add evaluator code-based` on a TTY opens its wizard. - llm-as-a-judge asks for a name, level, model, instructions and a rating scale preset. The model step offers Bedrock and OpenResponses, each opening its own model ID input. Bedrock is prefilled with global.anthropic.claude-sonnet-4-6, because the Evaluator service sends a temperature and newer Claude models refuse it. - code-based asks for a name, level and Lambda: scaffold a new one (with a timeout step) or use an existing one by ARN. - Each handler and its wizard build their input through one shared builder, so they refuse the same input with the same message. - RouterScreen keeps command names in one column when a description wraps. - The evaluator leaf descriptions follow the other add commands, and promptPreview moves into the wizard module for both reviews to use.

agentcore-devx-automation · 2026-09-30T00:36:57Z

Claude Security Review: no high-confidence findings. (run)

agentcore-devx-automation

AgentCore Harness Review

Verdict: Looks good

Nice split of the existing CLI handlers into toAdd...EvaluatorInput builders shared with the TUI wizards — the tests that reuse run(...) and compare against the wizard's outcome give strong evidence the two paths stay in sync. A few things I checked and found fine:

RevealChoiceField re-validates on return when the user commits, so per-provider stored model IDs (Bedrock vs OpenResponses) can't sneak an unvalidated value into the submit — only the currently-selected provider's value is used and it must be re-entered/validated to advance.
evaluatorNameSchema + requireDeployedNameFits runs live in the wizard and again in toAddCodeBasedEvaluatorInput / toAddLlmAsAJudgeEvaluatorInput, so the CLI path is still guarded.
The promptPreview move from harness/screen.tsx to components/wizard/fields.tsx (with its test relocated to wizard.test.tsx) is a clean refactor and the harness tests were updated to match.
DEFAULT_CODE_BASED_TIMEOUT_SECONDS now lives in the schema module and is used consistently by the template and the wizard default.
The RouterScreen two-Box layout fix has a dedicated regression test for wrapping on narrow terminals.

No blocking issues found. LGTM to merge.

codecov-commenter · 2026-09-30T00:40:25Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.26%. Comparing base (773cb64) to head (e0454ca).

Additional details and impacted files

@@             Coverage Diff              @@
##           refactor    #2468      +/-   ##
============================================
+ Coverage     97.24%   97.26%   +0.02%     
============================================
  Files           617      620       +3     
  Lines         43848    44241     +393     
============================================
+ Hits          42639    43033     +394     
+ Misses         1209     1208       -1

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…g the first line

agentcore-devx-automation · 2026-09-30T00:48:44Z

Claude Security Review: no high-confidence findings. (run)

nborges-aws · 2026-09-30T01:05:47Z

-
      const ratingScale = resolveRatingScale(flags["rating-scale"]);
-
      const resolver = new SourceResolver({ stdin: config.io.stdin });


Model validation happens after resolving --instructions. With --instructions -, an invalid --model waits for stdin instead of failing immediately. We should validate model/name before source I/O

Fixed in 4656785. The handler checks the deployed name and the model before it resolves --instructions, as it did before this PR. The wizard applies the same checks live on its name and model steps. A new test passes --instructions - with stdin left open. It timed out before the fix and now fails right away with the flag error.

nborges-aws · 2026-09-30T01:06:05Z

  );
 }

+export function promptPreview(prompt: string): string {


We should add back the length cap to this

Fixed in 4656785. The first line is capped at 60 characters again, followed by the line count. Tests cover a long single line and a long first line.

tejaskash

P1 — Path traversal in managed evaluator scaffolding
src/handlers/project/add/evaluator/code-based/index.ts:22
The headless --name accepts any nonempty string, while the managed branch never applies EvaluatorNameSchema. A name such as ../outside reaches filesystem path construction, can scaffold outside the project, and cleanup targets the wrong path. Validate the complete managed input before returning it. This vulnerability predates the PR but remains in the newly extracted validation boundary

- llm-as-a-judge checks the deployed name and the model before reading --instructions, so `--instructions -` no longer waits on stdin when a flag is already invalid. - The managed code-based branch validates its whole scaffold input against the evaluator schema before scaffolding. A --name such as ../outside, or a malformed --kms-key-arn, used to scaffold first and fail at the spec write, which could leave files outside the project. - promptPreview caps the first line at 60 characters again.

tejaskash · 2026-09-30T01:47:43Z

Re the path traversal finding: confirmed and fixed in 4656785.

I reproduced it with the real CLI. add evaluator code-based --name ../outside --level SESSION scaffolded a Lambda outside the project root, then failed at the spec write and left the files there. The same gap let a malformed --kms-key-arn through.

The managed branch now parses its whole scaffold input (name, level, description, KMS key, tags, timeout) against the evaluator schema before returning it. A bad name or KMS key now fails with exit code 2 and nothing on disk. A new test covers both and fails on the old code. The wizard was not affected, because its name step already applies EvaluatorNameSchema.

agentcore-devx-automation · 2026-09-30T01:50:14Z

Claude Security Review: no high-confidence findings. (run)

notgitika

LGTM but there is a merge conflict

…evaluator-tui # Conflicts: # src/components/RouterScreen.tsx

agentcore-devx-automation · 2026-09-30T02:21:46Z

Claude Security Review: no high-confidence findings. (run)

github-actions Bot added the size/xl PR size: XL label Sep 30, 2026

agentcore-devx-automation Bot added agentcore-harness-reviewing AgentCore Harness review in progress claude-security-reviewing Claude Code /security-review in progress labels Sep 30, 2026

agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Sep 30, 2026

agentcore-devx-automation Bot reviewed Sep 30, 2026

View reviewed changes

agentcore-devx-automation Bot removed the agentcore-harness-reviewing AgentCore Harness review in progress label Sep 30, 2026

tejaskash requested review from nborges-aws and notgitika September 30, 2026 00:39

github-actions Bot added size/xl PR size: XL and removed size/xl PR size: XL labels Sep 30, 2026

refactor(tui): show the line count in promptPreview instead of cuttin…

94e5585

…g the first line

github-actions Bot added size/xl PR size: XL and removed size/xl PR size: XL labels Sep 30, 2026

agentcore-devx-automation Bot added the claude-security-reviewing Claude Code /security-review in progress label Sep 30, 2026

agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Sep 30, 2026

nborges-aws requested changes Sep 30, 2026

View reviewed changes

tejaskash commented Sep 30, 2026

View reviewed changes

tejaskash requested a review from nborges-aws September 30, 2026 01:47

github-actions Bot added size/xl PR size: XL and removed size/xl PR size: XL labels Sep 30, 2026

agentcore-devx-automation Bot added the claude-security-reviewing Claude Code /security-review in progress label Sep 30, 2026

agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Sep 30, 2026

notgitika previously approved these changes Sep 30, 2026

View reviewed changes

nborges-aws previously approved these changes Sep 30, 2026

View reviewed changes

Merge remote-tracking branch 'origin/refactor' into feat/project-add-…

e0454ca

…evaluator-tui # Conflicts: # src/components/RouterScreen.tsx

tejaskash dismissed stale reviews from nborges-aws and notgitika via e0454ca September 30, 2026 02:17

notgitika approved these changes Sep 30, 2026

View reviewed changes

github-actions Bot added size/xl PR size: XL and removed size/xl PR size: XL labels Sep 30, 2026

agentcore-devx-automation Bot added the claude-security-reviewing Claude Code /security-review in progress label Sep 30, 2026

agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Sep 30, 2026

notgitika merged commit c845bee into refactor Sep 30, 2026
18 checks passed

notgitika deleted the feat/project-add-evaluator-tui branch September 30, 2026 02:25

Navigation Menu

feat(project): add LLM-as-a-judge and code-based evaluator TUI wizards - #2468

feat(project): add LLM-as-a-judge and code-based evaluator TUI wizards#2468
notgitika merged 4 commits into
refactorfrom
feat/project-add-evaluator-tui

tejaskash commented Sep 30, 2026 •

edited

Loading

Uh oh!

agentcore-devx-automation Bot commented Sep 30, 2026

Uh oh!

agentcore-devx-automation Bot left a comment

Uh oh!

codecov-commenter commented Sep 30, 2026 •

edited

Loading

Uh oh!

agentcore-devx-automation Bot commented Sep 30, 2026

Uh oh!

nborges-aws Sep 30, 2026

Uh oh!

tejaskash Sep 30, 2026

Uh oh!

nborges-aws Sep 30, 2026

Uh oh!

tejaskash Sep 30, 2026

Uh oh!

tejaskash left a comment

Uh oh!

tejaskash commented Sep 30, 2026

Uh oh!

agentcore-devx-automation Bot commented Sep 30, 2026

Uh oh!

notgitika left a comment

Uh oh!

agentcore-devx-automation Bot commented Sep 30, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants


		const ratingScale = resolveRatingScale(flags["rating-scale"]);

		const resolver = new SourceResolver({ stdin: config.io.stdin });

Conversation

tejaskash commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issue

Documentation PR

Type of Change

Testing

Uh oh!

agentcore-devx-automation Bot commented Sep 30, 2026

Uh oh!

agentcore-devx-automation Bot left a comment

Choose a reason for hiding this comment

AgentCore Harness Review

Uh oh!

codecov-commenter commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

agentcore-devx-automation Bot commented Sep 30, 2026

Uh oh!

nborges-aws Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

tejaskash Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

nborges-aws Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

tejaskash Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

tejaskash left a comment

Choose a reason for hiding this comment

Uh oh!

tejaskash commented Sep 30, 2026

Uh oh!

agentcore-devx-automation Bot commented Sep 30, 2026

Uh oh!

notgitika left a comment

Choose a reason for hiding this comment

Uh oh!

agentcore-devx-automation Bot commented Sep 30, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

tejaskash commented Sep 30, 2026 •

edited

Loading

codecov-commenter commented Sep 30, 2026 •

edited

Loading